Skip to content

v1.7: Postgres migration (WIP — tasks 1-4 done, 5 has open defects) - #6

Draft
NeverEndingCode wants to merge 14 commits into
mainfrom
worktree-v1.7-postgres-migration
Draft

v1.7: Postgres migration (WIP — tasks 1-4 done, 5 has open defects)#6
NeverEndingCode wants to merge 14 commits into
mainfrom
worktree-v1.7-postgres-migration

Conversation

@NeverEndingCode

@NeverEndingCode NeverEndingCode commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Status: in progress — do not merge, do not run against production data

Tasks 1–4 are complete and reviewed. Task 5 is built but has four unfixed defects. Tasks 6–8 are not started, which means the migrator itself does not exist yet.

Task State
1. Async db interface (292 call sites) ✅ complete, review clean
2. Facade + SQLite driver + schema split ✅ complete, 1 fix round
3. Postgres test harness + CI matrix ✅ complete, review clean
4. Postgres schema + driver ✅ complete, 1 fix round
5. identities auth split ⚠️ built, 4 Important findings unfixed
6. The SQLite → Postgres migrator ❌ not started
7. Auto-migrate on boot ❌ not started
8. Deployment config + docs ❌ not started

Current suite: SQLite 470/470, Postgres 467/470 + 3 legitimate skips, both backends green in CI matrix.

Documents

  • Design spec: docs/superpowers/specs/2026-08-01-postgres-supertokens-design.md
  • Implementation plan: docs/superpowers/plans/2026-08-01-v1.7-postgres.md
  • Backup / cutover / rollback runbook: docs/postgres-migration-runbook.md — the backup half is valid today

Task 5's open defects

All four are in the migration paths, not the application layer. The reviewer confirmed the username-collision retry logic, the provider subquery, and shared.js reuse are all correct.

  1. upsertUser writes users and identities non-atomically. A partial write leaves a users row with no identity. Every subsequent login then misses the identity lookup, retries INSERT INTO users on the same primary key, gets SQLSTATE 23505, misdiagnoses it as a username collision, and 500s — permanently, with no recovery path. This is the exact lockout class the retry logic exists to prevent.
  2. SQLite's foreign_key_check runs after COMMIT. A violation crashes one boot with the rebuild already durably committed; on restart the guard sees no provider column, returns early, and never checks again. Orphaned saves survive. The documented rebuild procedure runs the check inside the transaction so it can roll back.
  3. CREATE TABLE identities sits inside SQLite's guarded rebuild but outside Postgres's guard. On SQLite the table exists only as a side-effect of the users rebuild, so a database in a partially-migrated state boots clean and throws on first login. The Postgres path self-heals from the identical state.
  4. The base users DDL still declares provider/provider_id, which the migration then removes on every boot. users_new hardcodes a duplicate column list, so the next guardedAddColumn added in a future release would exist on users but be absent from users_new — silently losing that column's data on upgrade.

Landmines caught and fixed along the way

Found during implementation, not anticipated by the plan:

  • await x().prop parses as await (x().prop) — reads a property off a Promise, yields undefined rather than throwing. ~25 instances.
  • An ambient DATABASE_URL redirected the entire SQLite test run at a live database, writing to it and running DROP INDEX against it while reporting green — voiding the dual-backend guarantee with no signal. tests/helpers/backend.js is now the only writer of that variable.
  • A static import hoisting above process.env.DB_PATH, booting the driver against the on-disk database mid-test.
  • pg returns BIGINT as a string by default; every epoch-ms column here is BIGINT.
  • Postgres folds unquoted identifiers to lowercase, so listLeaderboard's AS userId would have returned userid and every rungsClaimed would read undefined on a leaderboard that still rendered.

Environment note

The dev machine has no docker binary — Podman 5.8.4 only. The test harness auto-detects the Podman socket and disables Ryuk (which cannot get the privileges it needs when rootless), while leaving the CI service-container path untouched. Task 8's docker compose verification steps will need Podman equivalents locally; the shipped docker-compose.yml remains correct for Unraid.

🤖 Generated with Claude Code

Evan Phyillaier and others added 11 commits August 1, 2026 12:15
Spec for moving persistence from SQLite to Postgres (v1.7) and adopting
SuperTokens without breaking existing Discord/GitHub logins (v1.8).

Decisions settled during brainstorming: dual backend with Postgres as the
default, tests against real Postgres in Docker, auto-migrate on boot behind
guards, and the identities auth split done up front so v1.8 stays small.

Two findings drove the design. SuperTokens supports external user ID mapping,
so users.id can stay 'provider:providerId' and no foreign key or
SUPER_ADMIN_IDS handling has to change. And SuperTokens' /auth/callback/github
is not a subdirectory of the currently registered /auth/github/callback, which
would fail GitHub's redirect_uri rule - the registered callback gets widened to
/auth so both paths work at once and passport keeps running throughout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight tasks, each ending in an independently testable deliverable:

1. Make the db interface async (SQLite unchanged underneath) - 292 call sites
2. Split db.js into facade + SQLite driver + schema module
3. Postgres test harness: Testcontainers, per-file databases, CI matrix
4. Postgres schema and driver, gated by a cross-dialect parity test
5. The identities auth split, with in-place upgrades for both backends
6. The SQLite to Postgres migrator, verified before COMMIT
7. Auto-migration on boot behind guards
8. Deployment config, Unraid runbook, smoke suites against Postgres

Task 1 is deliberately alone: mixing the async refactor with the driver split
would make the diff unreviewable.

Two landmines found while writing this that the spec had not caught, both
silent-corruption class rather than crash class:

- pg returns BIGINT as a string by default. Every epoch-ms column here is
  BIGINT and every consumer does arithmetic on it, so the int8 type parser
  has to be registered before the first query runs.
- Postgres folds unquoted identifiers to lowercase, so listLeaderboard's
  'AS userId' would return 'userid' and every rungsClaimed would read
  undefined on a leaderboard that still rendered.

Both are pinned by tests/db.parity.test.js.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every server/db.js export returns a Promise; all call sites across
server/ and tests/ now await. better-sqlite3 remains the implementation, so
behaviour is unchanged - this is purely the shape change that lets a pg
driver slot in behind the same interface.

shared/ is deliberately untouched and stays synchronous: the client bundles
it, and an async reducer would be a far larger change than this migration
needs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pure refactor - no SQL changes. server/db.js is now a re-export shim so every
existing import path keeps working, while server/db/ holds the seam the
Postgres driver plugs into next.

Adds a schema_migrations table: the schema has outgrown what CREATE TABLE IF
NOT EXISTS plus guarded ALTERs can honestly express, and the identities split
needs a real version marker.

Also extracts server/db/shared.js: findAvailableUsername, parseEventRow, and
the putEvent/upsertParticipation row normalizers are dialect-free logic the
Postgres driver will need identically, so they live once instead of being
duplicated (and drifting) per driver. findAvailableUsername is now async and
awaits its isTaken predicate - harmless for SQLite's synchronous predicate,
required for Postgres's async one - so one implementation serves both.
Static `import { driver } from '../server/db/index.js'` is hoisted by the
ESM spec above the `process.env.DB_PATH = ':memory:'` assignment earlier in
the same file, so it stood up the driver (and its data/rackstack.db file)
against the real default path instead of an in-memory DB - polluting a
real on-disk file across test runs and producing state leakage between
tests. Pull `driver` off the existing dynamic `dbMod` import instead, same
as every other export these files already consume that way.

Caught by rerunning tests/db.test.js in isolation during self-review: it
failed deterministically (config/roles state carried over from a prior
run) even though the full `npm test` run had passed, because npm test's
earlier passing run happened to be the one that created the stray file.
Code review finding: the function has been async (it awaits the shared,
async findAvailableUsername) since this refactor split it out of db.js, so
the inherited "Sync" suffix was false - misleading for Task 4's Postgres
driver author, who needs this exact seam. Renamed in schema.sqlite.js and
updated its one call site in applySchema. driver.sqlite.js imports it
aliased (dedupeUsernamesSchema) so it doesn't shadow the driver's own
dedupeUsernames interface method, which stays a thin, now explicitly
awaited, delegation to it (was a bare `return dedupeUsernamesSync(db)`,
inconsistent with the explicit awaits elsewhere in the same file).
Testcontainers locally, a service container in CI, and a per-test-file
database so the 19 db-touching suites cannot see each other's rows.

The matrix exists because we ship two backends: a SQLite driver that CI never
exercises would drift from the Postgres one, and dialect drift is the bug
class that loses data.

Locally, the only container runtime available is rootless Podman (no docker
binary), so pg-global.js points DOCKER_HOST at the Podman socket and disables
Ryuk when the developer hasn't already set either - Testcontainers then works
unmodified on Docker or Podman. Ryuk's absence makes teardown() the only
thing that stops/removes the container; StartedTestContainer#stop() removes
volumes too, so no reaper is required as long as teardown runs.

No Postgres driver exists yet (that's Task 4), so tests/harness.test.js is
the only suite that actually exercises TEST_DATABASE_URL right now - the
other 452 tests hardcode DB_PATH=':memory:' and keep running against SQLite
regardless of TEST_BACKEND.
Both backends now satisfy the same interface and the full suite runs against
each. tests/db.parity.test.js pins the behaviours that differ between the
dialects and would otherwise fail silently in production:

- pg returns BIGINT as a string by default; every epoch-ms column in this
  schema is BIGINT, so the int8 type parser is registered before any query.
- Postgres folds unquoted identifiers to lowercase, so listLeaderboard's
  camelCase aliases are double-quoted - unquoted, the client receives
  'userid' and every rungsClaimed reads undefined.
- config_history ordered by rowid on SQLite; Postgres has none, so it gains
  a BIGSERIAL key and the admin rollback UI keeps its newest-first order.
- COLLATE NOCASE becomes a unique functional index on lower(username).
- SQLITE_CONSTRAINT_UNIQUE becomes SQLSTATE 23505 - that retry path is what
  keeps a display-name collision from locking a player out on login.

Also points all 13 db-touching test files at provisionDatabase() instead of
hardcoding DB_PATH=':memory:', so npm test genuinely exercises Postgres
instead of silently falling back to SQLite regardless of TEST_BACKEND.
tests/db.test.js and tests/db.events.test.js gain Postgres-branch schema
introspection (pg_tables/information_schema vs. sqlite_master/PRAGMA); the
two guarded-ALTER tests are SQLite-only (schema.pg.js has no ALTER history
to replay) and skip on pg via it.runIf.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Critical: an ambient DATABASE_URL was never cleared on the sqlite test
path, and server/db/index.js checks it before DB_PATH - so a developer
with DATABASE_URL exported (plausible mid-migration, and the var Task 8
documents for production) would have every "sqlite" test file silently
connect to that real database instead. Centralized all env-var writing
for both backends inside provisionDatabase() itself (the pg branch is now
the only writer of DATABASE_URL in the codebase; the sqlite branch clears
it), removing the duplicated per-file if/else from all 14 test files.
Covered by a new regression test in tests/harness.test.js that sets
DATABASE_URL to an RFC-2606-reserved unreachable host and proves the
facade still resolves to sqlite end-to-end, plus a deliberate full
`npm run test:sqlite` run with that bogus DATABASE_URL exported (464/464
green in under 2s - no connection ever attempted).

Important: extracted dedupeUsernames' suffixing walk (identical in both
drivers apart from the SQL) into shared.js's dedupeUsernameRows, so the
-2/-3 convention can't drift between drivers again. Replaced the false
"exercised elsewhere in this suite" claim on db.test.js's guarded-ALTER
skip with a real cross-backend test that calls applySchema() a second
time against an already-populated database - the actual production
restart path - on both backends.

Minor: fixed db.parity.test.js's beforeAll/afterAll split (a RED-path
import failure left `db` undefined, so afterAll's TypeError masked the
real error and skipped cleanup) by switching to a top-level import like
every other rewired file; added ORDER BY tie-breakers to driver.pg.js's
listEvents/getAllUsersWithSaves so ties sort deterministically instead of
arbitrarily on Postgres.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
users.id stays 'provider:providerId', so all three foreign keys and
SUPER_ADMIN_IDS keep working untouched - the split is additive from the
application's point of view. upsertUser now resolves through identities and
records last_login_at.

Existing databases are migrated in place on boot. SQLite needs the full
table-rebuild dance because DROP COLUMN is refused while the column
participates in a table-level UNIQUE constraint, and the rebuild is followed
by a foreign_key_check - a violation there would mean orphaned saves. Both
schemas leave their original users CREATE TABLE untouched and migrate the
columns away as a guarded step that runs on every boot, so even a fresh
database exercises the real migration path, not just the dedicated
upgrade-path test.

getAllUsersWithSaves() keeps exposing provider via the user's primary
(earliest-created) identity - identical to the old column's output for
every single-identity user today. New listIdentities(userId) export is
unused until v1.8.

supertokens_user_id ships nullable and unused; v1.8 fills it in.
Written now rather than with Task 8 because the backup half is valid today and
should happen before anything else touches production data.

Leads with a status table making clear the migrator does not exist yet and
Task 5 has open defects, so nobody follows Part B against live data by mistake.

The rollback section names the one-way door explicitly: reverting to SQLite
always works mechanically, because the migration never modifies or deletes that
file, but it is frozen at cutover. Rolling back ten minutes later is free;
rolling back three days later silently discards three days of everyone's
progress with no error shown to anyone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@NeverEndingCode NeverEndingCode changed the title v1.7: Postgres migration + SuperTokens foundation (design) v1.7: Postgres migration (WIP — tasks 1-4 done, 5 has open defects) Aug 2, 2026
Evan Phyillaier added 3 commits August 2, 2026 00:37
…k, atomic upsertUser writes

Four Important findings from review, all in the migration paths:

1. identities is now created unconditionally on every boot, before the
   sqlite guard (schema.sqlite.js) - it was previously created only as a
   side effect of the users rebuild, so any database reaching "users has
   no provider column" without identities existing would boot cleanly and
   then throw on first login. Postgres already created it unconditionally.

2. schema.sqlite.js's foreign_key_check and its throw now run INSIDE the
   rebuild's db.transaction(), before it returns, so a violation aborts
   the rebuild instead of being discovered after COMMIT - previously the
   process died post-commit, but the next boot saw no provider column,
   skipped the rebuild, and never checked again: log-and-continue spread
   across two boots. The orphan test now also asserts users still has its
   provider column afterward, pinning the rollback rather than only the
   rejection.

3. upsertUser's two writes (users, identities) are now atomic on both
   drivers - db.transaction() on sqlite, a checked-out client with
   BEGIN/COMMIT/ROLLBACK on pg. Previously a failure on the second write
   left a users row with no identity, which the identity lookup can never
   find again - the next login attempt would retry INSERT INTO users on
   the same primary key and raise a constraint code neither retry guard
   recognizes, a permanent lockout. Covered by a new test that forces the
   identities write to fail and asserts no orphaned users row survives.

4. Both base `users` DDLs now ship in their final, post-split shape (no
   provider/provider_id) instead of declaring columns migrateIdentities
   immediately removes on every fresh boot. This closes the concrete risk
   that a future guardedAddColumn addition would exist on users but be
   absent from the rebuild's hardcoded users_new column list, silently
   dropping that column's data for any database still upgrading through
   the pre-split shape - now the rebuild only ever runs against a real
   upgrade target. users_new carries a comment warning it must stay in
   sync with the base DDL.

Plus the minors: foreign_keys restored in a finally; the pg guard pinned
to current_schema(); last_login_at set at insert (not just on return
visits); the upgrade-path test asserts save.data byte-for-byte; a case
pinning getAllUsersWithSaves returning provider: null for a user with no
identity row; migrateIdentities private on both backends; provider_id
added to the primary-identity tie-break.

npm run test:all: sqlite 472/472, pg 469/472 + 3 pre-existing skips.
The test added in 7cc2446 renamed `identities` away entirely to force
upsertUser's second write to fail. That also breaks the identity SELECT at
the top of upsertUser, so the miss branch's INSERT INTO users never even
ran - the test passed against both the fixed driver and the old, non-atomic
one, for the wrong reason (nothing to roll back either way).

Replaced with a trigger (SQLite: CREATE TRIGGER ... BEFORE INSERT; pg: a
BEFORE INSERT trigger function) that only fires for one specific
provider_id, so the identity lookup still misses normally, the users
insert still runs, and only the identities insert fails - the actual
failure mode the fix addresses.

Verified directly: reverted driver.sqlite.js and driver.pg.js to their
pre-fix (2f3bfe0) content in turn and reran this test - it now fails on
both, with the orphaned users row as the visible symptom, exactly as
described in the review. Restored the fixed drivers and reran - green on
both. Also reverted schema.sqlite.js's FK-check-inside-the-transaction fix
locally and confirmed the upgrade test's rollback-pinning assertion
(`expect(cols).toContain('provider')`) catches that regression too, before
restoring it.

npm run test:all: sqlite 472/472, pg 469/472 + 3 pre-existing skips.
…the vacuous last_login_at assertion

Two items from re-review:

1. Important (new, introduced by the prior fix round): moving schema.pg.js's
   base users DDL to the post-split shape (last round's Important 4) was
   correct, but it meant migrateIdentities's backfill-and-DROP-COLUMN branch
   became unreachable by any test that boots a driver normally - no pg test
   run ever exercised it anymore, unlike sqlite's dedicated old-shape test
   (which bypasses applySchema's DDL via a raw better-sqlite3 handle and so
   stayed unaffected). Reinstated as a permanent test: builds a genuinely
   pre-split users table directly against a fresh Postgres database (not via
   applySchema), runs applySchema over it, and asserts the columns are
   dropped, the identity backfilled, users.id unchanged, and the save
   preserved byte-for-byte - plus idempotency across two more calls. Gated
   on driver.__backend === 'pg' since no postgres container is even started
   when running the sqlite suite.

2. Minor 7 (still open): the returning-login test's last_login_at assertion
   was `>= ` against its own insert-time baseline, which is trivially true
   even with the UPDATE deleted entirely - the same vacuous-test shape
   caught in the atomicity test last round. Fixed by faking only `Date`
   (vi.useFakeTimers({ toFake: ['Date'] })), advancing the clock a real 50s
   between the two upsertUser calls, and asserting the exact resulting
   value - only true if the UPDATE actually ran and actually wrote it.

Both verified to fail against the un-fixed code before being finalized:
- Reverted schema.pg.js's backfill (kept the DROP COLUMN, removed the
  INSERT) -> new pg upgrade test fails with the identity row missing.
- Removed the `UPDATE identities SET last_login_at` statement from both
  drivers in turn -> the fake-timer assertion fails with the stale
  insert-time value on both backends.
All reverts restored before this commit; git diff against HEAD was empty
for every production file both times.

npm run test:all: sqlite 472/473 (1 skipped - pg-only test), pg 470/473
(3 skipped - pre-existing sqlite-only tests).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant